Device calibration via a projective transform matrix

ABSTRACT

A method is disclosed for improving accuracy of visual field testing in head-mounted displays. The method includes retrieving a visual field testing pattern for a head-mounted display, the visual field testing pattern including stimuli displayed at respective locations in a visual field of the head-mounted display. The visual field testing pattern is generated on the head-mounted display. Data is retrieved from a tilt sensor, located at the head-mounted display, for detecting degrees of head tilt of a user wearing the head-mounted display and the degree of head tilt is determined. A comparison is made between the degree of head tilt of the user to a first threshold degree. In response to the degree of head tilt of the user meeting or exceeding the first threshold degree, a recommendation to the user is generated for display on the head-mounted display.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a continuation-in-part of U.S. patent applicationSer. No. 17/246,054, entitled “Systems And Methods For Visual FieldTesting In Head-Mounted Displays,” filed Apr. 30, 2021, which is acontinuation of U.S. patent application Ser. No. 17/082,983, filed Oct.28, 2020, each of which is hereby incorporated by reference herein inits entirety.

BACKGROUND

Diagnosis of visual defects, such as blind spots, can be determined withconventional testing machines, such as a Humphry visual field analyzer.A patient is placed at the center of a curved portion of the analyzerand tests are performed by displaying images on the curved portion todetermine where the blind spots are located in the patient's visualfield. However, Humphry visual field analyzers as well as other testingmachinery is both expensive for wide distribution and requiresspecialized personnel for operating the machinery.

SUMMARY

Accordingly, systems and methods are disclosed herein for the use ofhead-mounted display devices and/or head-mounted display devices forvisual field testing. For example, these devices for visual fieldtesting lowers the costs related to performing visual field testing andimproves accessibility to visual field testing to a wider patient base.However, the adaption of visual field testing to these displays is notwithout its technical hurdles.

As a threshold technical problem, the introduction of visual fieldtesting into head-mounted display devices must account for the effectsof, or more accurately the lack thereof, of cyclotorsion. Cyclotorsionis the rotation of one eye around its visual axis. This rotation of theeye is what allows the visual field of a user to remain “right-side-up”even when the user tilts his or her head to one side or the other.However, as heads-up displays are fixed to the head of a user,cyclotorsion does not occur in the head-mounted display environment.That is, if a user tilts his or her head to one side or the other, thevisual field of the user tilts accordingly. Thus, the effects ofcyclotorsion present a threshold technical problem to overcome whenadapting introducing visual field testing into head-mounted displaydevices.

As described herein, one solution to overcoming the technical problemcaused by the differing effects of cyclotorsion in the head-mounteddisplay environment is to prevent a user from tilting his or her head.However, conventional optometry tools for preventing a user from tiltinghis or her head such as chin rests, or other structures built intooptometry equipment are ill-suited for a head-mounted displayenvironment. First, a requirement for a specialized structure ormodifications to head-mounted display devices negatively impacts theaccessibility of the devices as well as their ease of use. Second,specialized structures such as chin rests do not prevent any tiltingeffects caused by the head-mounted display devices being improperly wornand/or worn in a manner that introduces a slight tilt.

Accordingly, the systems and methods disclosed herein may usespecialized software and/or hardware elements implemented in thehead-mounted display devices to detect a tilting head of a user. Forexample, the head-mounted display device may include specialized sensorsand/or software used to interpret sensor data for the head-mounteddisplay device. The systems and methods may further generate alerts to auser based on detected head tilting and/or recommendations forcorrections of any head tilting. These alerts and recommendation mayfurther be presented on the head-mounted display to minimize the impactof head tilts during visual field testing.

As a supplementary technical problem, even when the differing effects ofcyclotorsion in the head-mounted display environment has been addressed,the adaption of visual field testing to head-mounted displays presents asecondary problem. Namely, visual field testing such as that performedby Humphry visual field analyzers is done by generating a series ofwhite light stimuli of varying intensities (brightness), throughout auniformly illuminated bowl. This illuminated bowl, or more precisely theillumination on a curved surface provides for standardized measurementsof vison from a center of fixation in terms of degrees. However,head-mounted display devices do not provide for surfaces with auniformed curvature. Instead, head-mounted display devices are generatedon flat surfaces and/or surfaces with non-uniformed curvature.Accordingly, light stimuli appearing on a head-mounted display mustaccount for these issues.

Methods, systems, and computer program products for improving accuracyof visual field testing in head-mounted displays are disclosed. In oneaspect, a method can include retrieving a visual field testing patternfor a head-mounted display, wherein the visual field testing patterncomprises icons that are displayed at respective locations in a visualfield of the head-mounted display. The method can also includegenerating for display the visual field testing pattern on thehead-mounted display; retrieving data from a tilt sensor, located at thehead-mounted display, for detecting degrees of head tilt of a userwearing the head-mounted display; determining, based on the dataretrieved from the tilt sensor, a degree of head tilt of the user;comparing, the degree of head tilt of the user to a first thresholddegree; and in response to the degree of head tilt of the user meetingor exceeding the first threshold degree, generating for display, on thehead-mounted display, a recommendation to the user.

Another technical problem in conventional head-mounted displays is thatcalibrating a head-mounted display needs to compensate for unknownsources of error that may affect assessment of the calibration. Forexample, eye tracking data received during calibration would be affectedif a head-mounted display was not being worn properly. This error (e.g.,when the head-mounted display is used for an eye examination) could thenbe interpreted as a visual defect rather than knowing it was caused byimproper wearing during calibration.

To address the above technical problems, the instant applicationdiscloses systems and methods that facilitate calibration of ahead-mounted display. For example, the system may generate calibrationpatterns for the user to view while the system tracks the eye movementof the user to determine what they are seeing. The analysis of the eyetracking data can generate a calibration for the head-mounted display aswell as a “calibration score” representing the accuracy of thecalibration. In this way, an optical practitioner will have both thebest possible calibration of a head-mounted display that may be used foreye examinations as well as a calibration score that may be indicativeof the accuracy of an eye test performed with the calibratedhead-mounted display.

Accordingly, methods, systems, and computer program products aredisclosed for calibrating head-mounted displays. One method forcalibrating a head-mounted display includes receiving edge eye trackingdata during edge calibration periods; calculating a projective transformmatrix based on the edge eye tracking data; receiving center eyetracking data during a center calibration period; applying theprojective transform matrix to the center eye tracking data to determinea gaze location; and generating a calibration score based on adifference between a center location and the gaze location.

The details of one or more variations of the subject matter describedherein are set forth in the accompanying drawings and the descriptionbelow. Other features and advantages of the subject matter describedherein will be apparent from the description and drawings, and from theclaims. While certain features of the currently disclosed subject matterare described for illustrative purposes in relation to particularimplementations, it should be readily understood that such features arenot intended to be limiting. The claims that follow this disclosure areintended to define the scope of the protected subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, show certain aspects of the subject matterdisclosed herein and, together with the description, help explain someof the principles associated with the disclosed implementations. In thedrawings,

FIG. 1A illustrates an example head-mounted display forming a wearabledevice for a subject in accordance with certain aspects of the presentdisclosure,

FIG. 1B illustrates a front view of the head-mounted display inaccordance with certain aspects of the present disclosure,

FIG. 1C an image of an example constructed head-mounted display inaccordance with certain aspects of the present disclosure,

FIGS. 1D-1E illustrate another example embodiment of a head-mounteddisplay, in accordance with certain aspects of the present disclosure,

FIG. 2 is a diagram illustrating correction of a visual field testingpattern by detecting and correcting for head tilt in accordance withcertain aspects of the present disclosure,

FIG. 3 is a diagram illustrating an exemplary method of accuratelyreplicating a visual field testing pattern from a curved surface on aflat surface in accordance with certain aspects of the presentdisclosure,

FIG. 4 is an illustrative system diagram for visual field testing usinga head-mounted display in accordance with certain aspects of the presentdisclosure,

FIG. 5 is a process flow diagram for correction of a visual fieldtesting pattern by detecting and correcting for head tilt in accordancewith certain aspects of the present disclosure,

FIG. 6 is a process flow diagram for accurately replicating a visualfield testing pattern from a curved surface on a flat surface inaccordance with certain aspects of the present disclosure,

FIG. 7 is a diagram illustrating an exemplary relationship between avirtual plane and a display plane as used to calibrate a head-mounteddisplay, in accordance with certain aspects of the present disclosure,

FIG. 8 is a diagram illustrating an exemplary central point and boundaryas used to generate a calibration score, in accordance with certainaspects of the present disclosure,

FIG. 9 is a process flow diagram for calibrating a head-mounted display,in accordance with certain aspects of the present disclosure, and

FIG. 10 is illustrative pseudocode for calibrating a head-mounteddisplay in accordance with certain aspects of the present disclosure.

DETAILED DESCRIPTION

The instant application describes systems and methods that facilitateperforming visual field testing, particularly utilizing worn gogglesthat provide testing patterns. One problem confronting optical carepractitioners is the effect of a patient tilting their head during eyeexaminations. If the head is tilted, this causes cyclotorsion, which isthe rotation of one eye around its visual axis. Uncorrected, this canintroduce error in an eye examination and misdiagnosis of opticalissues. In the art, a conventional diagnostic device used for testing isa Humphry visual field analyzer “Humphry analyzer.” Use of the Humphryanalyzer includes a patient placing their head at the center of asemispherical region with testing patterns projected at varyinglocations of the semispherical region. With the development of AugmentedReality (AR) and Virtual Reality (VR) goggles, similar testing can beperformed by projection of testing patterns upon the viewing surfaces ofsuch goggles. As referred to herein, embodiments may use a heads updisplay device or a head-mounted display device. For example, ahead-mounted display is a display device, worn on the head or as part ofa helmet, that may have a small display optic in front of one (monocularHMD) or each eye (binocular HMD).

One technical problem is the occurrence of cyclotorsion in patientsbeing tested using such goggles because while the goggles naturallyprovide compensation for head tilt, this only works if the goggles areworn properly (i.e., not tilted on the user's head). To address thisproblem, the instant application describes systems and methods fordetection and correction of goggle tilt relative to the user's head.Another technical problem is the display of accurate testing patternsusing the goggles, which have a flat viewing surface as compared to aHumphry analyzer, which has a curved viewing surface. To address thisadditional technical problem, methods are disclosed for generation oftesting patterns in goggles that are equivalent to those generated in aHumphry analyzer.

FIG. 1A illustrates an example head-mounted display 100 (e.g., goggles)forming a wearable device for a subject. In some embodiments, thehead-mounted display 100 may be a part of a visioning system asdescribed herein or in U.S. patent application Ser. No. 17/083,043,entitled “Vision Testing via Prediction-Based Setting of an InitialStimuli Characteristic for a User Interface Location” and filed Oct. 28,2020, the contents of which are hereby incorporated by reference in itsentirety. The head-mounted display 100 includes a left eyepiece 102 anda right eyepiece 104. Each eyepiece 102 and 104 may contain and/orassociate with a digital monitor configured to display (or project)recreated images to a respective eye of the subject. In variousembodiments, digital monitors may include a display screen, projectors,and/or hardware to generate the image display on the display screen. Itwill be appreciated that digital monitors comprising projectors may bepositioned at other locations to project images onto an eye of thesubject or onto an eyepiece comprising a screen, glass, or other surfaceonto which images may be projected. In one embodiment, the left eyepiece 102 and right eyepiece 104 may be positioned with respect to thehousing 106 to fit an orbital area on the subject such that eacheyepiece 102, 104 is able to collect data and display/project imagedata, which in a further example includes displaying/projecting imagedata to a different eye.

In some embodiments, each eyepiece 102,104 may further includes one ormore inward directed sensors 108, 110 may include infrared cameras,photodetectors, or other infrared sensors, configured to track pupilmovement and to determine and track visual axes of the subject. Theinward directed sensors 108, 110, e.g., comprising infrared cameras, maybe located in lower portions relative to the eye pieces 102, 104, so asto not block the visual field of the subject, neither their real visualfield nor a visual field displayed or projected to the subject. Theinward directed sensors 108, 110 may be directionally aligned to pointtoward a presumed pupil region for better pupil and/or line of sighttracking. In some examples, the inward directed sensors 108, 110 may beembedded within the eye pieces 102, 104 to provide a continuous interiorsurface. In some embodiments, head-mounted display 100 can include tiltsensor(s) 128 that can provide data on the degree of head tilt to aconnected computing system. As described further herein, the tiltsensors can be gyroscopes, water-based, etc.

FIG. 1B illustrates a front view of the head-mounted display 100,showing the front view of the eye pieces 102, 104, where respectiveoutward directed image sensors 112, 114 comprising field of visioncameras are positioned. In other embodiments, fewer or additionaloutward directed image sensors 112, 114 may be provided. The outwarddirected image sensors 112. 114 may be configured to capture continuousimages.

FIG. 1C is an image of an example constructed head-mounted display 100comprising eyepieces 102, 104 including two digital monitors, withfocusing lens 116, 118. In this example, only one inward directedoptical sensor 110 is included for pupil and line of sight tracking,however, in other examples, multiple inward directed optical sensors 110may be provided.

With respect to the FIGS. 1D-1E, an alternative embodiment ofhead-mounted 170 can include, in any combination, a high-resolutioncamera (or cameras) 102, a power unit 193, a processing unit 194, aglass screen 195, a see-through display 196 (e.g., a transparentdisplay), an eye tracking system 197, tilt sensor(s) 198 (similar totilt sensors 122), and other components.

In some examples, external sensors may be used to provide further datafor assessing visual field of the subject. For example, data used tocorrect the captured image may be obtained from external testingdevices, such as visual field testing devices, aberrometers,electro-oculograms, or visual evoked potential devices. Data obtainedfrom those devices may be combined with pupil or line of sight trackingfor visual axis determinations to create one or more modificationprofiles used to modify the images being projected or displayed to auser (e.g., correction profiles, enhancement profiles, etc., used tocorrect or enhance such images).

As used herein, when referring to the “head-mounted display,” even wherereference is made to the first embodiment (100), it is understood thatthe disclosed methods and operations apply to either head-mounteddisplay 100 or 170, unless specifically stated otherwise. It should benoted that, although some embodiments are described herein with respectto calibration of head-mounted displays, such techniques may be appliedfor calibration of one or more other user devices in other embodiments.

The head-mounted display 100 may be communicatively coupled with one ormore imaging processor through wired or wireless communications, such asthrough a wireless transceiver embedded within the head-mounted display100. An external imaging processor may include a computer such as alaptop computer, tablet, mobile phone, network server, or other computerprocessing devices, centralized or distributed, and may be characterizedby one or more processors and one or more memories. In the discussedexample, the captured images are processed in this external imageprocessing device; however, in other examples, the captured images maybe processed by an imaging processor embedded within the digitalspectacles. The processed images (e.g., enhanced to improve functionalvisual field or other vision aspects and/or enhanced to correct for thevisual field pathologies of the subject) are then transmitted to thehead-mounted display 100 and displayed by the monitors for viewing bythe subject.

The head-mounted display can be used to perform a visual assessments toidentify ocular pathologies, such as, high and/or low order aberrations,pathologies of the optic nerve such as glaucoma, optic neuritis, andoptic neuropathies, pathologies of the retina such as maculardegeneration, retinitis pigmentosa, pathologies of the visual pathway asmicrovascular strokes and tumors and other conditions such aspresbyopia, strabismus, high and low optical aberrations, monocularvision, anisometropia and aniseikonia, light sensitivity, anisocorianrefractive errors, and astigmatism.

In some examples, external sensors may be used to provide further datafor assessing visual field of the subject. For example, data used tocorrect the captured image may be obtained from external testing devicessuch as visual field testing devices, aberromaters, electro-oculograms,or visual evoked potential devices. Data obtained from those devices maybe combined with pupil or line of sight tracking for visual axisdeterminations to create the corrective profile of used to correct theimages being projected of displayed to the viewer.

The head-mounted display 100 may be communicatively coupled with one ormore imaging processor through wired or wireless communications, such asthrough a wireless transceiver embedded within the head-mounted display100. An external imaging processor may include a computer such as alaptop computer, tablet, mobile phone, network server, or other computerprocessing devices, centralized or distributed, and may be characterizedby one or more processors and one or more memories.

In an example operation of a vision system including the head-mounteddisplay, real-time image processing of captured images may be executedby an imaging processor, e.g., using a custom-built MATLAB (MathWorks,Natick, Mass.) code, that runs on a miniature computer embedded in thehead-mounted display. In other examples, the code may be run on anexternal image processing device or other computer wirelessly networkedto communicate with the head-mounted display.

FIG. 2 is a diagram illustrating correction of a visual field testingpattern by detecting and correcting for head tilt. As used herein, theterm “head tilt” refers to the angle between an axis of the head-mounteddisplay and an axis of the user's head. For example, such an angle maybe zero degrees when the head-mounted display 100 is worn correctly onthe user. In an embodiment, a system for improving accuracy of visualfield testing in head-mounted displays can include, in addition to thehead-mounted display, a tilt sensor for detecting degrees of head tiltof a user wearing the head-mounted display. In some cases, the tiltsensor can be located at the head-mounted display 100, though tiltsensors at other locations (e.g., external ones such as cameras thatview the user and the head-mounted display 100) are contemplated. Insome embodiments, the tilt sensor can be a water-based tilt sensor,similar to a level. In other embodiments, the tilt sensor canincorporate a gyro sensor or other types of rotation sensing hardware.

In the head-mounted display 100, or on an external computer, storagecircuitry can be configured to store and/or retrieve a visual fieldtesting pattern having stimuli (e.g., lights, patterns, icons,animations, etc.) that can be displayed at respective locations in thevisual field of the head-mounted display. There can also be controlcircuitry configured to generate for display the visual field testingpattern on the head-mounted display. Examples of a visual field testingpattern are shown in FIG. 2, with a fixation point 210 (typically nearthe center of the field of view) and a stimulus 220 that represents adisplayed stimulus for determining the location of a blind spot. The toppanel in FIG. 2 shows an example location of a blind spot (coincidentwith stimulus 220), e.g., as determined by the user being unable to seea stimulus displayed at that location. The middle panel illustrates theeffect of head tilt. Here, the head tilt causes the stimulus 220 to bedisplayed at a different location in the user's vision, outside of theblind spot 230. As a result, the blind spot may not be identified by theuser, possibly causing a misdiagnosis.

To address such issues, the system can determine, based on dataretrieved from the tilt sensor, a degree of head tilt of the user. Thedegree of head tilt can be determined, for example in the case of awater-based tilt sensor, the determination of water surface thatindicates the degree of tilt. One embodiment can include imaging a watersurface with miniaturized cameras to capture the water surface relativeto indicia that shows an un-tilted orientation. The angle between thewater surface and the indicia would then be the degree of head tilt.Another embodiment can include obtaining data from a plurality of watersensors (e.g., galvanic sensors) that are covered or exposed by waterdepending on the degree of tilt. The particular sensors detecting watercan then be used, such as via a lookup table, to determine the degree ofhead tilt. In some other embodiments, the degree of head tilt can bedetermined from received data from a gyroscope. The degree of head tiltof the user can be compared to a first threshold degree, such as 1, 2,5, 10, degrees, or any threshold as desired. The comparison itself caninclude one or more processors receiving the calculated degree of headtilt and performing a numerical comparison to the first thresholddegree. In response to the degree of head tilt of the user meeting orexceeding the first threshold degree, the system can generate fordisplay, on the head-mounted display, a recommendation to the user forreducing the head tilt. Such a recommendation can include a visualindication (e.g., red or green lights, a textual indication, etc.) thatthe head-mounted display 100 needs to be adjusted to remove the headtilt. The recommendation can include a display of the degree of headtilt in, for example, a graphical format (e.g., depicting an angle) ortextual format (e.g., the numerical value of the angle). Afteradjustment of the head-mounted display 100, testing can take place asshown in bottom panel of FIG. 2, showing that the stimulus remains atthe proper location for detecting the blind spot in the user's field ofvision.

In other embodiments, the system can automatically perform somecorrections, e.g., if the tilt is relatively small. Here, the controlcircuitry can be further configured to compare the degree of head tiltof the user to a second threshold degree (e.g., 0.1, 0.5, 1, 2 degrees,etc.) that is generally smaller than the first threshold degree. Such asecond threshold degree can be reflective of asymmetry in a user's facethat prevents perfect alignment, defects in the head-mounted display 100construction, small incidental tilts occurring during measurements, etc.The comparison of the degree of head tilt to the second threshold degreecan be performed in a manner similar to that described for the firstthreshold degree. In response to the degree of head tilt of the usermeeting or exceeding the second threshold degree, the system canautomatically adjust a respective location of the stimulus in the visualfield of the head-mounted display by a first amount. For example, if a0.1 degree tilt is detected, the system can automatically adjust thedisplay location of the icon to compensate by changing the coordinatesfor display of the stimulus to reflect the detected tilt. In this way,the first amount can be based on a distance of the stimulus from acenterpoint 240 of the visual field of the head-mounted display and adirection of the head tilt of the user. In some embodiments, centerpoint240 may correspond to a geometric center of the face of the head-mounteddisplay 100 and/or a center of fixation of the user. For example, insome embodiments, different head-mounted displays may have differentcenterpoints. Accordingly, the system may determine the centerpoint of ahead-mounted display and select respective locations of displayed iconsbased on the offset distance. For example, the system may determine acenterpoint of the head-mounted display based on receiving data from oneor more sensors. Additionally or alternatively, the system may receivesettings based on an initial calibration (e.g., an automatic calibrationor a manual calibration) when the system is activated. Additionally oralternatively, the system may input a model or serial number (or otheridentifier) for the head-mounted display into a look-up table listingcenterpoints for the model or serial number.

As shown in FIG. 2, centerpoint 240 can correspond to the center of thefixation point 210 and the direction 250 of the head tilt can be someangle (e.g., 10 degrees clockwise, 15 degrees, counterclockwise, etc.).Such a formulation permits a representation of the location of the iconrelative to the center point (e.g., {right arrow over (r)}=R cosθ{circumflex over (x)}+R sin θŷ), where {right arrow over (r)} is thevector from the center point to the icon having scalar distance R, whichis unchanged regardless of head tilt. The terms are directionalcomponents (e.g., x/y, horizontal/vertical) of the vector {right arrowover (r)} as a function of the head tilt angle θ. Thus, in anembodiment, the respective location of the icon can be defined by afirst directional component (e.g., a horizontal component) and a seconddirectional component (e.g., a vertical component). As shown in thebottom portion of FIG. 2, correction can include where the firstdirectional component is adjusted by a cosine of the degree of head tiltof the user and the second directional component is adjusted by a sineof the degree of head tilt. For example, the system can determine thedifference between the location of the icon before and after head tilt.This difference (for each directional component) can then be the amount(e.g., in pixels, cm, etc.) by which the respective location of the iconcan be adjusted.

FIG. 3 illustrates a simplified diagram depicting an exemplary method ofaccurately replicating a visual field testing pattern from a curvedsurface on a flat surface. Determining an angle of a visual defect canbe important in diagnosing and treating it. The Humphry analyzer, withits semispherical testing region 310, as depicted in FIG. 3 (top) canprovide a visual field testing pattern in the form of visual elements320 at angles of constant separation (e.g., 10, 20, 30, 40, etc.degrees). However, the head-mounted display 100 can have a flat surface330 (shown simplified and greatly enlarged, for illustrative purposes).If stimuli 340 are displayed at equidistant locations as shown, theywill not conform to the constant angular separation as described abovewith the Humphry analyzer, and thus not characterize the user's visionaccurately. Accordingly, the system may compensate for this difference.

Another consideration is that the offset distance (dimension b in thebottom of FIG. 3) between the flat surface where stimuli are displayedand the eye of the user can vary, based on the particular constructionof the head-mounted display 100, a user's facial structure, etc. Thisoffset distance can in turn affect where the stimuli 340 need to bedisplayed.

As shown in FIG. 3 (bottom), the disclosed methods allow respectivelocations of the stimuli 340 to be located in a row on the visual fieldand correspond to respective projections of points corresponding todifferent viewing angles along a curved surface onto a flat surface.This is depicted in FIG. 3 as can be seen by the stimuli 340, at theirrespective locations, being intersected by radial lines from visualelements 320. The respective locations can be determined based on anoffset distance of the head-mounted display and an angle to respectivepoints on the visual testing machine. The angle can be that referred toabove, (e.g., 10, 20, 30 degrees, etc.). Given the angle and the offsetdistance, the respective location corresponding to it is shown bydimension a, which is the distance from the center (e.g., 0 degrees) tothe respective location on flat surface 330. As one example, therespective locations can be determined based on the expression inEquation 1,

$\begin{matrix}{{a = {b\sqrt{\frac{1}{\cos^{2}\theta} - 1}}},} & (1)\end{matrix}$

where a is one of the respective locations, b is the offset distance,and θ is the angle.

While several simplifying assumptions have been taken for the purpose ofexplanation, it is understood that a person of skill would be able toincorporate variations in accordance with the present disclosure, forexample, accounting for the fact that each eye is off center (as opposedto the single viewing point assumed in FIG. 3), that the flat surfacemay indeed not be perfectly flat, but may contain some slight curvature(e.g., as depicted in FIGS. 1A and 1B), etc. Thus, as used herein, a“flat” surface is assumed to be the special case of a curved surfacehaving an infinite radius of curvature. As described in some portionsherein, the head-mounted display can have a finite radius of curvatureand thus be “curved” in the traditional sense.

In some implementations, the curvature of the head-mounted display canbe determined, and the respective locations selected, based on thecurvature. The determination of the curvature can be known or accessedbased on data from a known model of head-mounted display. Such curvaturevalues can be stored for retrieval or accessed via a network connection.The exact relation of how the presence of curvature affects the shiftingof the respective location is a function of the geometry of the system.Thus the disclosed methods contemplate a coordinate transformation fromthe intended angle θ to, for example, an analogous angle it, thatrepresents the angle along the curved surface of the head-mounteddisplay which would appear to the user to be at the intended angle.

Also, while the present disclosure has described visual field testingpatterns generally located on a horizontal “row,” it is contemplatedthat the disclosure applies to patterns that may be at an angle,vertical, or anywhere in a 2D plane. Similarly, such features can beextended to 3D visualizations, such as by altering the placement (andoptionally size) of the stimuli to give a depth effect, similar to aheads-up-display.

FIG. 4 is an illustrative system diagram for visual field testing usinga head-mounted display, in accordance with one or more embodiments. Forexample, system 400 may represent the components used to power thehead-mounted displays of FIGS. 1A-1C and perform the processes describedin FIGS. 5-6. As shown in FIG. 4, system 400 may include heads updisplay device 422 and user terminal 424. For example, heads up displaydevice 422 may be worn by a user, while progress of the user may bemonitored via user terminal 424. It should be noted that heads updisplay device 422 and user terminal 424 may be any computing device,including, but not limited to, a laptop computer, a tablet computer, ahand-held computer, other computer equipment (e.g., a server), including“smart,” wireless, wearable, and/or mobile devices. FIG. 4 may alsoinclude additional components such as cloud components 410. Cloudcomponents 410 may alternatively be any computing device as describedabove and may include any type of mobile terminal, fixed terminal, orother device. For example, cloud components 410 may be implemented as acloud computing system and may feature one or more component devices. Itshould also be noted that system 400 is not limited to three devices.Users, may, for instance, utilize one or more devices to interact withone another, one or more servers, or other components of system 400. Itshould be noted, that, while one or more operations are described hereinas being performed by particular components of system 400, thoseoperations may, in some embodiments, be performed by other components ofsystem 400. As an example, while one or more operations are describedherein as being performed by components of mobile device 422, thoseoperations, may, in some embodiments, be performed by components ofcloud components 410. In some embodiments, the various computers andsystems described herein may include one or more computing devices thatare programmed to perform the described functions. Additionally, oralternatively, multiple users may interact with system 400 and/or one ormore components of system 400. For example, in one embodiment, a firstuser and a second user may interact with system 400 using two differentcomponents.

With respect to the components of head-mounted display device 422, userterminal 424, and cloud components 410, each of these devices mayreceive content and data via input/output (hereinafter “I/O”) paths.Each of these devices may also include processors and/or controlcircuitry to send and receive commands, requests, and other suitabledata using the I/O paths. The control circuitry may comprise anysuitable processing, storage, and/or input/output circuitry. Each ofthese devices may also include a user input interface and/or user outputinterface (e.g., a display) for use in receiving and displaying data.For example, as shown in FIG. 4, both head-mounted display device 422and user terminal 424 include a display upon which to display data(e.g., a visual field test pattern).

It should be noted that in some embodiments, the devices may haveneither user input interface nor displays and may instead receive anddisplay content using another device (e.g., a dedicated display devicesuch as a computer screen and/or a dedicated input device such as aremote control, mouse, voice input, etc.). Additionally, the devices insystem 400 may run an application (or another suitable program). Theapplication may cause the processors and/or control circuitry to performoperations related to visual field testing.

Each of these devices may also include electronic storages. Theelectronic storages may include non-transitory storage media thatelectronically stores information. The electronic storage media of theelectronic storages may include one or both of (i) system storage thatis provided integrally (e.g., substantially non-removable) with serversor client devices, or (ii) removable storage that is removablyconnectable to the servers or client devices via, for example, a port(e.g., a USB port, a firewire port, etc.) or a drive (e.g., a diskdrive, etc.). The electronic storages may include one or more ofoptically readable storage media (e.g., optical disks, etc.),magnetically readable storage media (e.g., magnetic tape, magnetic harddrive, floppy drive, etc.), electrical charge-based storage media (e.g.,EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.),and/or other electronically readable storage media. The electronicstorages may include one or more virtual storage resources (e.g., cloudstorage, a virtual private network, and/or other virtual storageresources). The electronic storages may store software algorithms,information determined by the processors, information obtained fromservers, information obtained from client devices, or other informationthat enables the functionality as described herein.

FIG. 4 also includes communication paths 428, 430, and 432.Communication paths 428, 430, and 432 may include the Internet, a mobilephone network, a mobile voice or data network (e.g., a 5G or LTEnetwork), a cable network, a public switched telephone network, or othertypes of communications networks or combinations of communicationsnetworks. Communication paths 428, 430, and 432 may separately ortogether include one or more communications paths, such as a satellitepath, a fiber-optic path, a cable path, a path that supports Internetcommunications (e.g., IPTV), free-space connections (e.g., for broadcastor other wireless signals), or any other suitable wired or wirelesscommunications path or combination of such paths. The computing devicesmay include additional communication paths linking a plurality ofhardware, software, and/or firmware components operating together. Forexample, the computing devices may be implemented by a cloud ofcomputing platforms operating together as the computing devices.

Cloud components 410 may be a database configured to store user data fora user. For example, the database may include user data that the systemhas collected about the user through prior transactions. Alternatively,or additionally, the system may act as a clearing house for multiplesources of information about the user. Cloud components 410 may alsoinclude control circuitry configured to perform the various operationsneeded to generate recommendations. For example, the cloud components410 may include cloud-based storage circuitry configured to store afirst machine learning model that is trained to detect head tilt, adjustvisual testing patterns, and/or generate recommendations. Cloudcomponents 410 may also include cloud-based control circuitry configuredto determine an intent of the user based on a machine learning model.Cloud components 410 may also include cloud-based input/output circuitryconfigured to generate the dynamic conversational response during aconversational interaction.

Cloud components 410 includes machine learning model 402. Machinelearning model 402 may take inputs 404 and provide outputs 406. Theinputs may include multiple datasets such as a training dataset and atest dataset. Each of the plurality of datasets (e.g., inputs 404) mayinclude data subsets related to user data and visual testing patterns.In some embodiments, outputs 406 may be fed back to machine learningmodel 402 as input to train machine learning model 402 (e.g., alone orin conjunction with user indications of the accuracy of outputs 406,labels associated with the inputs, or with other reference feedbackinformation). For example, the system may receive a first labeledfeature input, wherein the first labeled feature input is labeled with atesting pattern adjustment for the first labeled feature input. Thesystem may then train the first machine learning model to classify thefirst labeled feature input with the known testing pattern adjustment.

FIG. 5 is a process flow diagram for correction of a visual fieldtesting pattern by detecting and correcting for head tilt. For example,process 500 may represent the steps taken by one or more devices, asshown in FIGS. 1A-1C, when providing visual field testing using ahead-mounted display.

At step 502, process 500 (e.g., using one or more components in system400 (FIG. 4)) retrieves a visual field testing pattern for ahead-mounted display. For example, the system may retrieve a visualfield testing pattern for a head-mounted display, wherein the visualfield testing pattern comprises stimuli that are displayed at respectivelocations in a visual field of the head-mounted display. In anotherexample, the respective location of the icon can be defined by a firstdirectional component and a second directional component. The firstdirectional component can be adjusted by a cosine of the degree of headtilt of the user and the second directional component can be adjusted bya sine of the degree of head tilt of the user.

In yet another example, the respective locations of the stimuli can belocated in a row on the visual field and the respective locations cancorrespond to respective projections of points corresponding todifferent viewing angles along a curved surface onto a flat surface.Also, in other examples, the respective locations can be determinedbased on an offset distance of the head-mounted display and an angle torespective points on the visual testing machine. Accordingly, in someexamples, the respective locations are determined based on theexpression

${a = {b\sqrt{\frac{1}{\cos^{2}\theta} - 1}}},$

where a is one of the respective locations, b is the offset distance,and θ is the angle.

At step 504, process 500 (e.g., using one or more components in system400 (FIG. 4)) generate for display the visual field testing pattern. Forexample, the system may generate for display the visual field testingpattern on the head-mounted display.

At step 506, process 500 (e.g., using one or more components in system400 (FIG. 4)) retrieves data from a tilt sensor. For example, the systemmay retrieve data from a tilt sensor for detecting degrees of head tiltof a user wearing the head-mounted display. The tilt sensor can be, forexample, located at the head-mounted display.

At step 508, process 500 (e.g., using one or more components in system400 (FIG. 4)) determines a degree of head tilt of a user. For example,the system may determine, based on the data retrieved from the tiltsensor, a degree of head tilt of the user.

At step 510, process 500 (e.g., using one or more components in system400 (FIG. 4)) compare the degree of head tilts. For example, the systemmay compare, using the control circuitry, the degree of head tilt of theuser to a first threshold degree. In another example, process 500 cancompare the degree of head tilt of the user to a second threshold degreeand in response to the degree of head tilt of the user meeting orexceeding the second threshold degree, automatically adjusts arespective location of a stimulus of the plurality of icons in thevisual field of the head-mounted display by a first amount. For example,the first amount can be is based on a distance of the icon from acenterpoint of the visual field of the head-mounted display and adirection of the head tilt of the user.

At step 512, process 500 (e.g., using one or more components in system400 (FIG. 4)) generate a recommendation to the user. For example, thesystem may generate for display a recommendation to the user. Forexample, the recommendation can be displayed on the head-mounteddisplay. The generation can also be in response to the degree of headtilt of the user meeting or exceeding the first threshold degree.

It is contemplated that the steps or descriptions of FIG. 5 may be usedwith any other embodiment of this disclosure. In addition, the steps anddescriptions described in relation to FIG. 5 may be done in alternativeorders or in parallel to further the purposes of this disclosure. Forexample, each of these steps may be performed in any order, in parallel,or simultaneously to reduce lag or increase the speed of the system ormethod. Furthermore, it should be noted that any of the devices orequipment discussed in relation to FIGS. 1-3 could be used to performone or more of the steps in FIG. 5.

FIG. 6 is a process flow diagram for accurately replicating a visualfield testing pattern from a curved surface on a flat surface. Forexample, process 600 may represent the steps taken by one or moredevices, as shown in FIGS. 1A-1C, when providing visual field testingusing a head-mounted display.

At step 602, process 600 (e.g., using one or more components in system400 (FIG. 4) retrieves a visual field testing pattern for a head-mounteddisplay. For example, the system may retrieve a visual field testingpattern for a head-mounted display, wherein the visual field testingpattern comprises stimuli that are displayed at respective locations ina visual field of the head-mounted display.

At step 604, process 600 (e.g., using one or more components in system400 (FIG. 4)) determines a curvature of the head-mounted display. Forexample, the system may determine a curvature of the head-mounteddisplay based on receiving data from one or more sensors. Additionallyor alternatively, the system may receive settings based on an initialcalibration (e.g., an automatic calibration or a manual calibration)when the system is activated. Additionally or alternatively, the systemmay input a model or serial number (or other identifier) for thehead-mounted display into a look-up table listing curvatures for themodel or serial number.

Additionally or alternatively, in some embodiments, the system maydetermine an offset distance of the head-mounted display based onreceiving data from one or more sensors. Additionally or alternatively,the system may receive settings based on an initial calibration (e.g.,an automatic calibration or a manual calibration) when the system isactivated indicating the offset distance. Additionally or alternatively,the system may input a model or serial number (or other identifier) forthe head-mounted display into a look-up table listing offset distancefor the model or serial number.

At step 606, process 600 (e.g., using one or more components in system400 (FIG. 4)) selects the respective locations based on the curvature.For example, the system may automatically adjust the respectivelocations based on the curvature and/or offset distance determined bythe system. In some embodiments, the system may receive the curvatureand/or offset distance (e.g., via input entered into a user terminal(e.g., user terminal 424 (FIG. 4)) and adjust the respective locationsaccordingly.

At step 608, process 600 (e.g., using one or more components in system400 (FIG. 4)) generates for display the visual field testing pattern onthe head-mounted display. For example, in generating the visual fieldtesting pattern, the respective locations of the stimuli can be locatedin a row on the visual field. In another example, the respectivelocations can correspond to respective projections of pointscorresponding to different viewing angles along a curved surface onto aflat surface.

It is contemplated that the steps or descriptions of FIG. 6 may be usedwith any other embodiment of this disclosure. In addition, the steps anddescriptions described in relation to FIG. 6 may be done in alternativeorders or in parallel to further the purposes of this disclosure. Forexample, each of these steps may be performed in any order, in parallel,or simultaneously to reduce lag or increase the speed of the system ormethod. Furthermore, it should be noted that any of the devices orequipment discussed in relation to FIGS. 1-3 could be used to performone or more of the steps in FIG. 6.

FIG. 7 illustrates a simplified diagram depicting an exemplaryrelationship between a virtual plane and a display plane as may be usedto calibrate a head-mounted display. As described further herein, thedisclosed systems may generate a calibration pattern comprising a numberof stimuli (e.g., one or more graphical elements, referred to herein as“icons”) for display at the head-mounted display.

The calibration may take many forms and may comprises one or morestimuli being displayed in series and/or in parallel. The system maydisplay a pattern in which the stimuli are displayed at particularpositions. The positions may be defined by the system in terms of aheight, width and/or viewing angle. The system may generate the stimuliat the extremes of the visual field in order to achieve the bestcalibration. For example, the system may display the stimuli in one ormore corners of the visual field in order to receive the bestmeasurement for calibrating a user's gaze location on a single fixationpoint (e.g., a centerpoint in the visual field).

The example of FIG. 7 depicts a simplified representation showing aviewing plane 730 having a number of stimuli (e.g. edge stimuli 732,734, 736, 738) that may be seen by a user. However, for example due toimproper wearing or other errors, eye tracking data obtained from a userviewing any one of these edge stimuli may not correspond to where theeye tracking data would be expected based on where the edge stimulus isdisplayed by the head-mounted display. Example eye tracking data 710 isshown corresponding to edge stimulus 734. The eye tracking data isdepicted as a dashed line representing the path of the eye over itsacquisition time. In this example, even though the system generates edgestimulus 734, the eye tracking data generally surrounds the perceivededge point 724.

In some embodiments, the system may retrieve calibration data of a giveninterval. The use of the given interval allows the system to normalizedata during this time to remove outlier that may occur as a naturalresult of the calibration process. For example, the edge point can bedetermined by receiving eye tracking data over periods of time referredto herein as edge calibration periods. The edge calibration periods maybe, for example, one second, five seconds, etc. In some implementations,eye tracking data may be averaged over such periods of time to generatean average location. This process may be repeated for a number of edgestimuli, with four shown in the example of FIG. 7. Thus, as depicted,edge points 732, 734, 736, and 738 generate corresponding points 722,724, 726, and 728. While four points are shown, in other embodiments,other numbers of points may be used, for example, three, five, eight,etc. in some embodiments, the system may generate edge stimuli on edgesof the field of view of the head-mounted display. For example, along oneor more of the left, right, upper, or lower edges. In certainembodiments, edge stimuli may be generated at the corners of the field,for example, upper left, upper right, lower left, or lower right. It isalso contemplated that stimuli may be generated anywhere within thefield of view stimuli may be generated anywhere within the field of viewsuch that the presently described calibrations may be performed.

As part of various technical solutions that address the disclosedshortcomings of conventional calibration methods, certain disclosedembodiments describe how the system may relate and assess differencesbetween what is displayed and what the user sees. Similar to how theedge stimuli may define a display plane 730 (i.e., a plane establishedby the system where the edge stimuli are displayed on), the edge pointsmay also define a virtual plane 720. The system may calculate aprojective transform matrix based on the edge eye tracking data thatconverts any location in virtual plane 720 to display plane 730. Thus,as described further below, stimuli or other calibration patterns may begenerated by the head-mounted display and the obtained eye tracking datamay be mapped back onto the display plane for comparison with thecalibration pattern.

The system may calculate a projective transform matrix that isespecially useful for a general transformation (e.g., one that does notforce parallelism to be observed as such may not be the case whenformerly parallel stimuli are viewed by a person). The below exampleillustrates how the system may generate and/or utilize a projectivetransform matrix for a coordinate transformation between the two planes720 and 730:

${\begin{pmatrix}a_{1} & a_{2} & b_{1} \\a_{3} & a_{4} & b_{2} \\c_{1} & c_{2} & 1\end{pmatrix}\begin{pmatrix}x \\y \\1\end{pmatrix}} = {\begin{pmatrix}x^{\prime} \\y^{\prime} \\1\end{pmatrix}.}$

In the above matrix equation, the 2×2 “a” submatrix is a rotationmatrix, the 2×1 “b” submatrix is a translation vector, and the 1×2 “c”submatrix is a projection vector. The x,y elements correspond to the x,ycoordinates of the edge stimulus in the display plane (e.g., edgestimulus 734) and the x′,y′ elements corresponding to the x,ycoordinates of the point in the virtual plane (e.g., point 724). Toapply the projective transform metric, the system may execute pseudocodesuch as shown in FIG. 10.

For example, the projective transformation can be represented astransformation of an arbitrary quadrangle (i.e. system of four points)into another one. Alternatively or additionally, the system may use atransform based on a different number of points. For example, the systemmay use an affine transformation, which is a transformation of atriangle. The system may select the type of transform based on thenumber of stimuli generated. The system may select the number of stimuligenerated based on one or more criteria. For example, the system maydetermine a number of stimuli needed to achieve a determined amount ofaccuracy and/or meet a particular threshold level of precision. Thesystem may likewise select the number of stimuli based on a type oftest, amount of calibration needed, and/or a frequency of calibration.

For example, the system may determine that a four point (e.g.,projective transform) calibration is used at the initiation of the userof a head mounted device. The system may then determine (e.g., using ahierarchy of criteria) whether an additional calibration needs to beperformed, and if so, how many stimuli are required to be displayed.

FIG. 8 illustrates a simplified diagram depicting an exemplary centralpoint and boundary as used to generate a calibration score. In someimplementations, it may be of more interest to base a calibration off ofa point more central in a field of view, as such is where most visualtesting patterns are displayed. The system may generate for display acenter stimulus 800 on the head-mounted display at a center location.This system may also receive center eye tracking data during a centercalibration period, similar to the edge calibration period(s) describedabove. The calculated point in the virtual plane may be transformed towhat is referred to herein as a “gaze location” in the display plane bythe system utilizing the projective transform matrix. The system maygenerate a calibration score based on a difference 840 (e.g., a delta inpixels, mm, or other similar distance metrics) between the centerstimulus 800 and the gaze location 810. The inset shows this example ingreater detail and includes exemplary eye tracking data 830 and thedifference 840 between center stimulus 800 and gaze location 810. Also,in various embodiments the difference 840 may be similarly calculatedthe virtual plane 720 via a determination of equivalent points for thecenter stimulus and gaze location. In this way, is contemplated that anycombination of points may be utilized in either plane and related toeach other via the projective transform matrix to calculate differences,locations relative to a boundary (as described below), etc.

In some implementations, the system may assess the accuracy of thecalibration based on whether the gaze location and/or eye tracking datais within a prescribed boundary. For example, as shown in FIG. 8, thesystem may generate boundary 820. In some implementations, such aboundary may be a circle having a given radius from the center stimulus800, but other boundary shapes such as square, hexagonal, etc. may beused. While in some implementations the boundary may be visuallydisplayed by the head-mounted display, this is not necessary and insteadthe boundary may merely reside as coordinates or other boundary definingalgorithm in computer memory. Accordingly, the system may determine thecalibration score based on the size (e.g. radius) of the boundary.

As previously mentioned, the calibration score may be indicative of theconfidence in the calibration. In this way, should a calibration bedetermined by the system to fail (e.g., the gaze location being outsidethe radius of the boundary), the system may repeat at least a portion ofthe calibration (e.g., the acquiring of edge eye tracking data, centereye tracking data, and/or calculation of the projective transformmatrix), but making the size of the boundary larger (e.g., a largerradius boundary). For example, in one embodiment if the gaze location iscalculated to be within the first (or initial) boundary generated, thecalibration may be assigned a score of 100 (perhaps corresponding to thebest possible calibration). If that calibration attempt were to havefailed, then, for example, the radius of the boundary may be increased20% and if that calibration succeeds then it may be assigned a score of90. Any such relationship between boundary size and calibration scoremay be used by the system, as implemented by a person of skill.

The system may also determine, based on the gaze location, whether auser is looking at the center location. Such a determination may be madeby the system, for example, if the gaze location is outside of abounding box, an area defined by the edge stimuli, etc. Another relatedimplementation can further refine the calibration is not allowing large,sustained deviations in the eye tracking data, even if the averagelocation is within one of the above-described boundaries. For example,the system may determine that the user is not looking at the centerlocation based on whether at least a portion of the center eye trackingdata deviates from the gaze location more than a spatial deviationthreshold and for longer than a temporal deviation threshold. As onespecific example, the spatial deviation threshold may be any distanceoutside the boundary, but may also be a larger boundary (e.g., 1.1×,1.5× the radius of the present boundary). While a brief excursion may beallowed, the temporal deviation threshold may be set by the system tobe, for example, 1 ms, 10 ms, 100 ms, etc. In this way, the system woulddetermine that the calibration failed if the user's gaze drifted, forexample, far to the left and stayed there, indicating a possible loss offocus or attention on the calibration process.

FIG. 9 is a process flow diagram for calibrating a head-mounted display.For example, process 900 may represent steps taken by one or moredevices, as shown in FIGS. 1A-1E, when calibrating the head-mounteddisplay.

At step 902, process 900 (e.g., using one or more components in system400 (FIG. 4)) receives edge eye tracking data. For example, the systemmay receive edge eye tracking data during edge calibration periods.Additionally or alternatively, the system may generate for display anumber of edge stimuli on the head-mounted display. Additionally oralternatively, the system may generate edge stimuli on edges or of afield of view of the head-mounted display.

At step 904, process 900 (e.g., using one or more components in system400 (FIG. 4)) calculates a projective transform matrix. For example, thesystem may calculate a projective transform matrix based on the edge eyetracking data. For example, the system may use pseudocode 1000 (FIG. 10)and/or the process described in FIG. 7.

At step 906, process 900 (e.g., using one or more components in system400 (FIG. 4)) receives center eye tracking data. For example, the systemmay receive center eye tracking data during a center calibration period.Additionally or alternatively, the system may generate for display acenter stimulus on the head-mounted display at the center location.Additionally or alternatively, the system may generate a boundary aroundthe center stimulus and may display the boundary at the head display.

At step 908, process 900 (e.g., using one or more components in system400 (FIG. 4)) applies a projective transform matrix to the center eyetracking data. For example, the system may apply the projectivetransform matrix to the center eye tracking data to determine a gazelocation.

At step 910, process 900 (e.g., using one or more components in system400 (FIG. 4)) generates a calibration score. For example, the system maygenerate a calibration score based on a difference between a centrallocation and the gaze location. The calibration score may be indicativeof the accuracy of an eye test performed with the head-mounted display.Additionally or alternatively, the system may determine, based on thedifference, whether the gaze location is inside the boundary.Additionally or alternatively, the calibration score may be based on asize of the boundary. Additionally or alternatively, the system, inresponse to the difference indicating that the gaze location is outsidethe boundary, may repeat at least a portion of the calibration, whereinthe size of the boundary is larger. Additionally or alternatively, thesystem may determine whether a user is looking at the center locationbased on the gaze location. In response to the determination that theuser is not looking at the center location, the system may repeat atleast a portion of the calibration. Additionally or alternatively, thedetermination by the system that the user is not looking at the centerlocation may require that at least a portion of the center eye trackingdata deviates from the gaze location more than a spatial deviationthreshold and for longer than a temporal deviation threshold.

It is contemplated that the steps or descriptions of FIG. 9 may be usedwith any other embodiment of this disclosure. In addition, the steps anddescriptions described in relation to FIG. 9 may be done in alternativeorders or in parallel to further the purposes of this disclosure. Forexample, each of these steps may be performed in any order, in parallel,or simultaneously to reduce lag or increase the speed of the system ormethod. Furthermore, it should be noted that any of the devices orequipment discussed in relation to FIGS. 1A-1E could be used to performone or more of the steps in FIG. 9.

FIG. 10 is illustrative pseudocode for calibrating a head-mounteddisplay in accordance with certain aspects of the present disclosure.For example, pseudocode 1000 represents illustrative pseudocode forcalculating a projective transform matrix as described herein. The belowexample illustrates how the system may generate and/or utilize aprojective transform matrix for a coordinate transformation between thetwo planes (e.g., planes 720 and 730 of FIG. 7). For example, pseudocode1000 may generate values for 4×4 described in FIG. 7. In such cases, thepoints identified in pseudocode 1000 may correspond to the x,ycoordinates of the edge stimulus in the display plane (e.g., edgestimulus 734) and x,y coordinates of the point in the virtual plane(e.g., point 724).

The above-described embodiments of the present disclosure are presentedfor purposes of illustration and not of limitation, and the presentdisclosure is limited only by the claims which follow. Furthermore, itshould be noted that the features and limitations described in any oneembodiment may be applied to any other embodiment herein, and flowchartsor examples relating to one embodiment may be combined with any otherembodiment in a suitable manner, done in different orders, or done inparallel. In addition, the systems and methods described herein may beperformed in real time. It should also be noted that the systems and/ormethods described above may be applied to, or used in accordance with,other systems and/or methods.

In the following, further features, characteristics, and exemplarytechnical solutions of the present disclosure will be described in termsof items that may be optionally claimed in any combination:

1. A method, the method comprising: retrieving a visual field testingpattern for a head-mounted display; and generating for display thevisual field testing pattern on the head-mounted display.2. The method of any of the preceding items, wherein the visual fieldtesting pattern comprising stimuli that are displayed at respectivelocations in a visual field of the head-mounted display.3. The method of any of the preceding items, further comprisingretrieving data from a tilt sensor, located at the head-mounted display,for detecting degrees of head tilt of a user wearing the head-mounteddisplay; determining, based on the data retrieved from the tilt sensor,a degree of head tilt of the user; and comparing, the degree of headtilt of the user to a first threshold degree.4. The method of any of the preceding items, further comprisinggenerating for display, on the head-mounted display, a recommendation tothe user in response to the degree of head tilt of the user meeting orexceeding the first threshold degree.5. The method of any of the preceding items, further comprising:comparing the degree of head tilt of the user to a second thresholddegree; and in response to the degree of head tilt of the user meetingor exceeding the second threshold degree, automatically adjusting arespective location of a stimulus of the stimuli in the visual field ofthe head-mounted display by a first amount.6. The method of any of the preceding items, wherein the first amount isbased on a distance of the stimulus from a centerpoint of the visualfield of the head-mounted display and a direction of the head tilt ofthe user.7. The method of any of the preceding items, wherein the respectivelocation of the stimulus is defined by a first directional component anda second directional component, and wherein the first directionalcomponent is adjusted by a cosine of the degree of head tilt of the userand the second directional component is adjusted by a sine of the degreeof head tilt of the user.8. The method of any of the preceding items, wherein the respectivelocations of the stimulus are located in a row on the visual field, andwherein the respective locations correspond to respective projections ofpoints corresponding to different viewing angles along a curved surfaceonto a flat surface.9. The method of any of the preceding items, wherein the respectivelocations are determined based on an offset distance of the head-mounteddisplay and an angle to respective points on the visual testing machine.10. The method of any of the preceding items, wherein the respectivelocations are determined based on the expression

${a = {b\sqrt{\frac{1}{\cos^{2}\theta} - 1}}},$

where a is one of the respective locations, b is the offset distance,and q is the angle.11. The method of any of the preceding items, further comprisingdetermining a curvature of the head-mounted display and selecting therespective locations based on the curvature.12. The method of any of the preceding items, further comprisingdetermining an offset distance of the head-mounted display and selectingthe respective locations based on the offset distance.13. The method of any of the preceding items, further comprisingdetermining a centerpoint of the head-mounted display and selecting therespective locations based on the centerpoint.14. A method for calibrating a head-mounted display, the methodcomprising: receiving edge eye tracking data during a plurality of edgecalibration periods; calculating a projective transform matrix based onthe edge eye tracking data; receiving center eye tracking data during acenter calibration period; applying the projective transform matrix tothe center eye tracking data to determine a gaze location; andgenerating a calibration score based on a difference between a centerlocation and the gaze location.15: The method of Item 14, further comprising: generating for displaystimuli on the head-mounted display; and generating for display a centerstimulus on the head-mounted display at the center location.16: The method of Item 14 or any of the preceding items that dependtherefrom, wherein the stimuli are generated on edges of a field of viewof the head-mounted display.17: The method of Item 14 or any of the preceding items that dependtherefrom, wherein the stimuli are generated at corners of a field ofview of the head-mounted display.18: The method of Item 14 or any of the preceding items that dependtherefrom, wherein the calibration score is indicative of the accuracyof an eye test performed with the head-mounted display.19: The method of Item 14 or any of the preceding items that dependtherefrom, further comprising: generating a boundary around the centerstimulus; determining, based on the difference, whether the gazelocation is inside the boundary; and determining the calibration scorebased on a size of the boundary.20: The method of Item 14 or any of the preceding items that dependtherefrom, further comprising displaying the boundary at the headmounted display.21: The method of Item 14 or any of the preceding items that dependtherefrom, further comprising: in response to the difference indicatingthat the gaze location is outside the boundary, repeating at least aportion of the calibration, wherein the size of the boundary is larger.22: The method of Item 14 or any of the preceding items that dependtherefrom, further comprising: determining whether a user is looking atthe center location based on the gaze location; and in response to thedetermination that the user is not looking at the center location,repeating at least a portion of the calibration.23: The method of Item 14 or any of the preceding items that dependtherefrom, wherein the determination that the user is not looking at thecenter location requires that at least a portion of the center eyetracking data deviates from the gaze location more than a spatialdeviation threshold and for longer than a temporal deviation threshold.24: A system for calibrating head-mounted displays, the systemcomprising: a head-mounted display; inward directed sensors, located atthe head-mounted display, configured to track pupil movement; storagecircuitry configured to store a plurality of icons that are displayed atrespective locations in a visual field of the head-mounted display; andcontrol circuitry configured to: to perform operations comprising thoseof any of items 1-23.25: A tangible, non-transitory, machine-readable medium storinginstructions that, when executed by a data processing apparatus, causethe data processing apparatus to perform operations comprising those ofany of items 1-23.26. A system comprising: one or more processors; and memory storinginstructions that, when executed by the processors, cause the processorsto effectuate operations comprising those of any of items 1-23.27. A system comprising means for performing any of items 1-23.

The present disclosure contemplates that the calculations disclosed inthe embodiments herein may be performed in a number of ways, applyingthe same concepts taught herein, and that such calculations areequivalent to the embodiments disclosed.

One or more aspects or features of the subject matter described hereincan be realized in digital electronic circuitry, integrated circuitry,specially designed application specific integrated circuits (ASICs),field programmable gate arrays (FPGAs) computer hardware, firmware,software, and/or combinations thereof. These various aspects or featurescan include implementation in one or more computer programs that areexecutable and/or interpretable on a programmable system including atleast one programmable processor, which can be special or generalpurpose, coupled to receive data and instructions from, and to transmitdata and instructions to, a storage system, at least one input device,and at least one output device. The programmable system or computingsystem may include clients and servers. A client and server aregenerally remote from each other and typically interact through acommunication network. The relationship of client and server arises byvirtue of computer programs running on the respective computers andhaving a client-server relationship to each other.

These computer programs, which can also be referred to programs,software, software applications, applications, components, or code,include machine instructions for a programmable processor, and can beimplemented in a high-level procedural language, an object-orientedprogramming language, a functional programming language, a logicalprogramming language, and/or in assembly/machine language. As usedherein, the term “machine-readable medium” (or “computer readablemedium”) refers to any computer program product, apparatus and/ordevice, such as for example magnetic discs, optical disks, memory, andProgrammable Logic Devices (PLDs), used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” (or “computer readable signal”)refers to any signal used to provide machine instructions and/or data toa programmable processor. The machine-readable medium can store suchmachine instructions non-transitorily, such as for example as would anon-transient solid-state memory or a magnetic hard drive or anyequivalent storage medium. The machine-readable medium can alternativelyor additionally store such machine instructions in a transient manner,such as for example as would a processor cache or other random accessmemory associated with one or more physical processor cores.

To provide for interaction with a user, one or more aspects or featuresof the subject matter described herein can be implemented on a computerhaving a display device, such as for example a cathode ray tube (CRT) ora liquid crystal display (LCD) or a light emitting diode (LED) monitorfor displaying information to the user and a keyboard and a pointingdevice, such as for example a mouse or a trackball, by which the usermay provide input to the computer. Other kinds of devices can be used toprovide for interaction with a user as well. For example, feedbackprovided to the user can be any form of sensory feedback, such as forexample visual feedback, auditory feedback, or tactile feedback; andinput from the user may be received in any form, including, but notlimited to, acoustic, speech, or tactile input. Other possible inputdevices include, but are not limited to, touch screens or othertouch-sensitive devices such as single or multi-point resistive orcapacitive trackpads, voice recognition hardware and software, opticalscanners, optical pointers, digital image capture devices and associatedinterpretation software, and the like.

In the descriptions above and in the claims, phrases such as “at leastone of” or “one or more of” may occur followed by a conjunctive list ofelements or features. The term “and/or” may also occur in a list of twoor more elements or features. Unless otherwise implicitly or explicitlycontradicted by the context in which it used, such a phrase is intendedto mean any of the listed elements or features individually or any ofthe recited elements or features in combination with any of the otherrecited elements or features. For example, the phrases “at least one ofA and B;” “one or more of A and B;” and “A and/or B” are each intendedto mean “A alone, B alone, or A and B together.” A similarinterpretation is also intended for lists including three or more items.For example, the phrases “at least one of A, B, and C;” “one or more ofA, B, and C;” and “A, B, and/or C” are each intended to mean “A alone, Balone, C alone, A and B together, A and C together, B and C together, orA and B and C together.” Use of the term “based on,” above and in theclaims is intended to mean, “based at least in part on,” such that anunrecited feature or element is also permissible.

The subject matter described herein can be embodied in systems,apparatus, methods, computer programs and/or articles depending on thedesired configuration. Any methods or the logic flows depicted in theaccompanying figures and/or described herein do not necessarily requirethe particular order shown, or sequential order, to achieve desirableresults. The implementations set forth in the foregoing description donot represent all implementations consistent with the subject matterdescribed herein. Instead, they are merely some examples consistent withaspects related to the described subject matter. Although a fewvariations have been described in detail above, other modifications oradditions are possible. In particular, further features and/orvariations can be provided in addition to those set forth herein. Theimplementations described above can be directed to various combinationsand subcombinations of the disclosed features and/or combinations andsubcombinations of further features noted above. Furthermore, abovedescribed advantages are not intended to limit the application of anyissued claims to processes and structures accomplishing any or all ofthe advantages.

Additionally, section headings shall not limit or characterize theinvention(s) set out in any claims that may issue from this disclosure.Further, the description of a technology in the “Background” is not tobe construed as an admission that technology is prior art to anyinvention(s) in this disclosure. Neither is the “Summary” to beconsidered as a characterization of the invention(s) set forth in issuedclaims. Furthermore, any reference to this disclosure in general or useof the word “invention” in the singular is not intended to imply anylimitation on the scope of the claims set forth below. Multipleinventions may be set forth according to the limitations of the multipleclaims issuing from this disclosure, and such claims accordingly definethe invention(s), and their equivalents, that are protected thereby.

What is claimed is:
 1. A system for calibrating head-mounted displaysutilizing a projective transform matrix, the system comprising: ahead-mounted display; inward directed sensors, located at thehead-mounted display, configured to track pupil movement; storagecircuitry configured to store a plurality of icons that are displayed atrespective locations in a visual field of the head-mounted display; andcontrol circuitry configured to: generate for display a plurality ofedge icons on the head-mounted display; receive edge eye tracking dataduring a plurality of edge calibration periods corresponding to thedisplay of the plurality of edge icons; calculate the projectivetransform matrix based on the edge eye tracking data; generate fordisplay a center icon on the head-mounted display at a center location;receive center eye tracking data during a center calibration period;apply the projective transform matrix to the center eye tracking data todetermine a gaze location; and generate a calibration score based on adifference between the center location and the gaze location.
 2. Amethod for calibrating a head-mounted display, the method comprising:receiving edge eye tracking data during a plurality of edge calibrationperiods; calculating a projective transform matrix based on the edge eyetracking data; receive center eye tracking data during a centercalibration period; applying the projective transform matrix to thecenter eye tracking data to determine a gaze location; and generating acalibration score based on a difference between a center location andthe gaze location.
 3. The method of claim 2, further comprising:generating, for display, edge stimuli on the head-mounted display; andgenerating, for display, a center stimulus on the head-mounted displayat the center location.
 4. The method of claim 3, wherein the edgestimuli are generated on edges of a field of view of the head-mounteddisplay.
 5. The method of claim 4, wherein the edge stimuli aregenerated at corners of a field of view of the head-mounted display. 6.The method of claim 3, wherein the calibration score is indicative ofthe accuracy of an eye test performed with the head-mounted display. 7.The method of claim 3, further comprising: generating a boundary aroundthe center stimulus; determining, based on the difference, whether thegaze location is inside the boundary; and determining the calibrationscore based on a size of the boundary.
 8. The method of claim 7, furthercomprising displaying the boundary at the head mounted display.
 9. Themethod of claim 7, further comprising: in response to the differenceindicating that the gaze location is outside the boundary, repeating atleast a portion of the calibration, wherein the size of the boundary islarger.
 10. The method of claim 3, further comprising: determiningwhether a user is looking at the center location based on the gazelocation; and in response to the determination that the user is notlooking at the center location, repeating at least a portion of thecalibration.
 11. The method of claim 10, wherein the determination thatthe user is not looking at the center location requires that at least aportion of the center eye tracking data deviates from the gaze locationmore than a spatial deviation threshold and for longer than a temporaldeviation threshold.
 12. A tangible, non-transitory, machine-readablemedium storing instructions that, when executed by a data processingapparatus, cause the data processing apparatus to perform operationscomprising: receiving edge eye tracking data during a plurality of edgecalibration periods; calculating a projective transform matrix based onthe edge eye tracking data; receiving center eye tracking data during acenter calibration period; applying the projective transform matrix tothe center eye tracking data to determine a gaze location; andgenerating a calibration score based on a difference between a centerlocation and the gaze location.
 13. The machine-readable medium of claim12, further comprising: generating, for display, edge stimuli on thehead-mounted display; and generating, for display, a center stimulus onthe head-mounted display at the center location.
 14. Themachine-readable medium of claim 13, wherein the edge stimuli aregenerated on edges of a field of view of the head-mounted display. 15.The machine-readable medium of claim 14, wherein the edge stimuli aregenerated at corners of a field of view of the head-mounted display. 16.The machine-readable medium of claim 13, further comprising: generatinga boundary around the center stimulus; determining, based on thedifference, whether the gaze location is inside the boundary; anddetermining the calibration score based on a size of the boundary. 17.The machine-readable medium of claim 16, further comprising displayingthe boundary at the head mounted display.
 18. The machine-readablemedium of claim 16, further comprising: in response to the differenceindicating that the gaze location is outside the boundary, repeating atleast a portion of the calibration, wherein the size of the boundary islarger.
 19. The machine-readable medium of claim 13, further comprising:determining whether a user is looking at the center location based onthe gaze location; and in response to the determination that the user isnot looking at the center location, repeating at least a portion of thecalibration.
 20. The machine-readable medium of claim 19, wherein thedetermination that the user is not looking at the center locationrequires that at least a portion of the center eye tracking datadeviates from the gaze location more than a spatial deviation thresholdand for longer than a temporal deviation threshold.